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The primary outcome of any information feedback 
system must be data for the classroom teacher. For this reason, a 
system's value has to be measured in terms of its usefulness to the 
teacher in making instructional decisions. A report should contain 
only data that the teacher needs «ind should be produced in an 
understandable format. Oser feedback should be solicited to modify 
the report design. Adequate time should be allowed in the first 
production run for debugging. Once the system is perfected, 
arrangements may be made with the data processing operations 
personnel to institute an automatic procedure which will prepare the 
input, run the job, and distribute the output. However, it is 
desirable to have someone familiar with the output to examine it for 
quirks before distribution. Some of the current productions of the 
Office of Research and Evaluation of the School District of 
Philadelphia provide examples of useful teacher reports. One report, 
for the diagnosis of pupil performance, lists each pupil's answer to 
each test item, showing at a glance the student's strengths and 
weaknesses on a given test or subject. (Author/SL) 
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Frances R» Byers 
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The primary outcome of any Information feedbock system must be data for 
the classroom teacher. For this reason, a system's value has to be measured 
in terms of how useful It is in helping the teacher make Instructional 
decisions about her pupils and thereby facilitate the educative process. If 
the element of usabi I i ty (and its correlate, unders tandabl 1 1 ty) Is lacking, 
the information provided (no matter how accurate or potentially valuable) wfll 
be shunted aside as just another piece of useless paper, handed down by '^the 
administration," which will not help the teacher one Iota in dealing with the 
problems she must confront each day in her classroom* Aside from being useful, 
an information feedback system must be efficient, If efficiency Is lacking, 
the teacher receives the information too late to be of use In her Instructional 
plans. She needs help in deciding how to place her pupils, In determining which 
topics she should stress to correct weakness In her pupils' skills, or in plan- 
ning an effective sequence of topics for a newly-Implemented course of study, 
D esign of the System-Usabil i ty and UnderstandabI 1 I ty 

In order to determine what constitutes useful teacher Information, a 
sincere effort must be made to find out what kinds of information the teachers 
actually desire. Although it seems almost too obvious a requirement to mention, 
very often computerized reports are fashioned In such a way that the user finds 
the information to have little immediate value, This happens particularly when 
'^output" is produced for users other than the teacher (usually program or dis- 
trict administrators or others who require data such as summary statistics 
across classrooms or schools, data upon which to make policy decisions). 



Therefore, care must be taken to provide <>n1y those data that teacher can 
nctu.illy use: or stated d i f f eron( ly , the output must bo tailored to ntoct the 
needs of the user . 

Once the content of a teacher's report Is decided upon, the report must 
be produced In an unders tandabl e format. The terminology and data must be 
presented in a way both easy to comprehend and pleasing to look at. Wherever 
possible^ English words or commonly understood abbreviations should be used 
instead of obstruse numerical codes. One way to Insure communication Is to 
receive affirmative answers to the following questions; Are the numbers (scores* 
ratings, category codes, etcj presented so that their meaning Is readily under- 
stood? Are the elements of data spaced sufficiently on the page so as to pre- 
vent confusion and/or ey^.s train? (Sometimes a positive answer to these quest lon'^i 
cannot be assured until the next aspect In the development of the Information 
system, described below, is carried out.) 

Another way to promote usability Is to recognize that systems design fs not 
just a one-time thing. It involves an ongoing relationship between the systems 
designer and/or programmer and school/field representat i ves i The system's program 
(or series of programs) Is actually being pilot-tested during Its first production 
runs. It is being tested in the sense that, for that Initial period of time, the 
users of the system*s product are getting acquainted with It and sensing what, 
for them, are the good and bad aspects of that product. They then should forward 
critical feedback to the systems designer as to how the product can be improved, 
One should be prepared to make whatever modifications are feasible and reasonable. 
Implementation of the S ystem-Ef ficiency 

Initial product ion runs . In a way, this aspect may be the most frustrating 
tn the entire sequence of a system^s development. When a new prograr-^ Is : 



developed, the oporatlonol dendltne for Us products Is often close on \hv, heels 
of itr. dcvelopmentcil phuse, This moans that the program may not have hod odequni 
testing and debugging prior to the time when It ts called upon to produce 
reports. So the first several "production*' runs are mass-data test runs durtng 
which many unfores'^.en proble.T^s surface. Therefore, one should (1) plan to set 
.-islde sufficient time for full-scale testing as well as (2) allow for a greater 
"turnaround'* period than would be scheduled for the regular system. Problems 
always occur so It Is best to allow extra time for them. It Is the very rare 
Instance Indeed when a system or even a single program encounters full success 
during Its first production run(s). 

Ongoing product i on runs > Once the system's programs are "perfected" to the 
point where major modifications are no longer needed, an arrangement should be 
made with the data processing operations personnel to Institute an "automatic" 
procedure which will prepare the Input, run the Job, and distribute the output. 
Once in operation, the systems designer and/or programmer need only be called 
upon in an emergency situation. Even then, however. It is advisable to have 
someone familiar with the output examine It before It Is distributed to the 
schools, Even with thoroughly debugged programs, unexpected quirks In the data 
(possibly produced by erroneous coding or keypunching) can produce unexpected 
quirks In the output. (So says the old maxim; Garbage in, Garbage out.) To 
maintain the credibility of the ^>ystem, such quirks should be spotted before 
distribution and accounted for, perhaps with a note of explanation to the 
offending user so that the same mistakes will be avoided In the future. 

To summarize, the major points In the design of the system are: 

1. Produce only the information needed, 

2 . Produce 1 t in an understandable format . 



3. Receive feedback as to its acceptability and suggestions for 
improvement , 

^. Modify lha system to conform with reasonable user suqqestions. 
And the major points in the Implementation of the system are: 

1, Allow adequate testing time and simulate actual production runs as 
c losel y as poss i bl e . 

2, Mal^e appropriate arrangements for the regular running of the system 
programCs) with the personnel in the operations area of the data 
process ing Instal lat ion< 

3, Arrange for the examination of the output before distribution so 
that ''garbage'* output is not allowed to be distributed wfthout some 
Indication as to the problem, 

Some Examples of Useful Teacher I nformation Reports 

Refer to Figures lA, B, C, and 0 for abbreviated examples of some teacher 
reports currently being produced by the Office of Research and Evaluation of 
The School District of Philadelphia. 

The report displayed in Figure lA represents one of our most widely-used 
types of computer reports. Because the report Is for the diagnosis of pupil 
performance, the computer program (in FORTRAN IV) has a number of features. 
Test items can be divided into subtests of varying length (as indicated on the 
pupil listing by the columns of stars following Items Ifi, 27 and 31. One can 
see, In the legend in the upper left-hand corner of each page, that subtest 1 
of the Sight and Sound Inventory, Form A , comprises items 1 to 18, initial 
consonants). Each pupiPs total number-right score is listed on both pages of 
the class 1 i st 1 ng . 

in addition, a listing is made of each pupil's answer to each Item, 
abbreviated to one or two characters. For example, if a pupil answers an item 
correctly, a is printed; If he answers the Item incorrectly, the correct 
■ answer Is printed, Therefore, the report helps the teacher tel 1 at a qlance^^^^ ; 



(reading across tho sheet) which Items o ijlven child has answered corrf^ctly 
and incorrectly (yieldinq Information of a diagnostic nature about the rhllrl's 
strengths and weaknesses In the content covered by the tost)» It also gives 
the teacher Information about (reading down the sheot) which Items gave her 
pupils the most trouble (yielding Information of a currlcular nature about 
what areai, of the content need to receive greater stress In Instruction). 
Below the class listing, tho teacher Is given the number and percent of pupils 
in her class who answered each item correctly and the mean score for the class. 

It should be noted that ^^subtest mastery" can be set before the program 
is run to Indicate whether each examinee has effectively mastered the content 
of each subtest. In the Figure !A example, mastery Is defined as having no 
more than one Incorrect response per subtest., (That is, at least 17 items 
must be correct out of 18 items in subtest one, eight correct out of nine In 
subtest two, and three correct out of four in subtest three.) A teacher can 
make both diagnostic and currlcular judgments about how well her pupils mastered 
each subtest by looking across and down, respectively. In addition, these class 
data are summarized as the number and percent of the class mastering each sub- 
test (see bottom entry). 

In Figure IB, a class listing for the expanded version of the Sight and 
Sound Inventory, Form B, with six subtests as explained fn the legend at the 
top, one can tell that the class as a whole had particular trouble with the 
short-E (SE) vowel sound (36 percent answering correctly) and the AW, 01, and 
OU sounds (answered correctly by 28, 2k, and 32 percent, respectively). On 
the other hand, 80 percent of the class effectively mastered subtest two on 
digraphs , and 8^ percent of the class effectively mastered subtest three on 
thr^e-letter combinations. Future i ns t ruct ion wou Id focus nrimar My on the 
content covered by subtest five, long and short vowel sounds, because 



only percent mastered the I tem<> of that subtest, and subtest six, other 
vowel sounds, nustered by only 16 percent. On the diagnostic level, one can 
SCO thni whMi? at )e«ist 60 percent of the class rmistorod the first four subtests, 
pupils MAN and MIL need parHcuhir help In those areas, and pupil HOD needs help 
In finol sounds. 

In Figure IC, a class listing of the Secondary Group Headinq Inventory^ 
the items are divided Into six subtests of (supposedly) Increasing difficulty. 
(If such is the case, one may ask why only 23 percent of the class mastered the 
first level, but this question reflects on the quality of the test, not on the 
reporting system.) Here the Items are indicated, not by their correct answer, 
but by their type: F for factual, V for vocabulary, and 1 for Inferential. In 
subtest one> item 7, an inferential Item, posed particular difficulty only 
seven percent of the examinees (one pupil) answered it correctly. The same Is 
true for an increasing number of Items in successive subtests, until no one 
mastered ihe last two subtests. One can examine each pupIHs performance to 
fearn who needs what kind of help. At the third reading level (subtest 1), . 
there are a number of pupils who need help even with factual or vocabulary 
Items. Special Instruction may be needed to supplement these pupils' sklMs 
i n these areas . 

In Figure ID, the Arithmetic Concepts subtest of the Iowa Tests of Basic 
Skills, is shown that the items again are designated by type (C for Currency, 
etc.), but an indication of subtest mastery Is not produced. Interpretation of 
the report can still be made on an Item-by-ltem basis, as Just described. 

Although the program producing the above described reports has been in use for 
about two years I It has undergone several major modi flea clons 1 n response to teacher 
and supervisor requests, An exar^ple Is the addition of the subtest mastery feature , 
Even now, In spite of the fact that i ts use I s widespread and the teachers seem to 
use it enthusiastically, I have a list of at least a half-do2en suggested improve- 
ments which will be made in the near future, 
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